Systems, methods, and computer program products for expediting expertise

ABSTRACT

A method includes generating, as executed by a processor on a computer, a plurality of topic-specific user knowledge models for each user of a plurality of users, each topic-specific user knowledge model representing a level of knowledge possessed by a respective user on a single topic from a set of globally defined topics shared among the plurality of users, generating a plurality of topic-specific expert knowledge models, each topic-specific expert knowledge model representing an aggregate level of knowledge possessed by a plurality of expert users on a single topic from a set of globally defined topics shared among the plurality of users, comparing the topic-specific user knowledge model of the first user with the topic-specific expert knowledge model for a respective topic to determine a distance between a user knowledge level and an aggregate expert knowledge level for the topic.

The present application is a continuation application of U.S.application Ser. No. 13/648,988, filed on Oct. 10, 2012, the entirecontent of which is incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention generally relates to systems, methods, andcomputer program products for expediting a level of expertise of atopic, and more particularly to expediting a level of expertise of atopic based on the activity of a plurality of users as that activityrelates to content.

2. Description of the Related Art

Corporations have a constant need for increasing the expertise of theiremployees in an efficient manner in order to improve enterpriseproductivity and profitability. The learning needed to increaseexpertise can be either formal (e.g., structured classroom or onlinecourses) or informal (e.g., self-directed learning activities such asreading articles). On the one hand, there is a long and rich traditionof providing formal training solutions, but they are largely inflexibleand require the learners to follow a fixed curriculum path, which isdifficult if not impossible to customize to suit the needs and interestsof any particular individual.

On the other hand, despite some conventional attempts at providing asolution to this problem, few focus on providing structured support tothe informal learning that takes place naturally as part of everydaybusiness. When it comes to informal learning, the burden is placed onindividual learners as individual learners must still select suitablelearning materials from the vast amount of available content, therebyresulting in an extremely difficult task that often results in thelearners giving up before they reach their target level of expertise.Moreover, those learners who do not give up are given no feedbackregarding a comparison between their current expertise level and theirtarget level of expertise. As a result, learners are often leftquestioning their progress (if any), thereby resulting in even furtherburnout.

Furthermore, individual learners may not know who the domain experts areand hence may resort to asking questions to any number of individuals.As a result, it is possible that the individual learner will receiveresponses from those who may not be recognized as an expert in a fieldrelevant to the individual learner's question. In such a case, theindividual learner can become further confused by inaccurate responsesto their questions.

Therefore, there is a need for learning systems, methods and computerprogram products that are customizable, flexible, able to providefeedback regarding a comparison between an expertise of an individuallearner and that of their target level of expertise, and able to detectpotential experts for a set of users.

SUMMARY OF THE INVENTION

In view of the foregoing and other exemplary problems, drawbacks, anddisadvantages of the conventional methods and structures, an exemplaryfeature of the present invention is to provide learning systems, methodsand computer program products that are customizable, able to providefeedback regarding a comparison between an expertise of an individuallearner and that of their target level of expertise, and able to detectpotential experts for a set of users.

A first aspect of the present disclosure allows for an expeditedincrease on a number of expert users within an enterprise. Activities ofusers who are wishing to become experts in a topic may be analyzed and arecommendation of activities may be proposed based on the activities ofexperts for that topic, thereby expediting a user's ability to engage inthe same activities of an expert. As a result, when compared toconventional technologies, this aspect of the present disclosure allowsfor a faster acceleration of expertise within an enterprise.

A second exemplary aspect of the present disclosure allows user who areemerging experts to be ascertained based on their activities. Similarly,emerging topics to be ascertained and, as a result, a user or anenterprise can to respond more quickly to changing market needs.

A third exemplary aspect of the present disclosure facilitates expertiseramp-up through the use of user modeling and expertise modeling. Thecontent and activities associated with a given user is analyzed andupdated based on available information from various sources (e.g.employee learning tools, online social business tools, paper and patentdatabases, organizational structure, and other information repositories)to create a user model of topics in which this user is well versed.Further, the expertise model for a given topic may be created byaggregating individual user models of identified experts on given topic.A user's distance and path to the target level of expertise for a giventopic are determined by comparing the model of the user for the topicand the expertise model for the topic. Both the user models and theexpertise models are may be updated by incorporating new content anduser activities, thereby allowing the dynamic evaluation and monitoringof the expertise levels of specific individual users, as well as, theaggregate of all contributors including the experts.

A fourth exemplary aspect of the present disclosure includes usingunstructured data/content created in an organic way (e.g., through theactivities and the content associated with those activities of bothexperts and non-experts) to create topic, user, and expertise models.These models can by created, for example, by using an automaticmining/clustering algorithm to process activities and associatedcontent. According to at least this exemplary aspect, the presentdisclosure can support applications for an aggregate population (e.g.,identify gaps in an enterprise) in addition to individual users, etc. toprovide better support for informal learning in a social environment.

Another exemplary aspect of the present disclosure includes a systemincluding a global topic model building module that builds a globaltopic model based on a set of data that is updated according to anactivity of each user of a plurality of users, the global topic modelincluding a topic representation for a topic; a user model buildingmodule that builds a plurality of user models, each user model beingbuilt based on an activity of a respective user; an expertise modelbuilding module that builds an expertise model for the topic based onthe activity of at least one user of the plurality of users, theexpertise model for the topic setting a target level of knowledge for afirst user of the plurality of users; a processor that executesinstructions to compare a user model of the first user with theexpertise model for the topic; and an expertise assessment and learningrecommendation module that recommends an activity associated with theset of data to the first user based on the comparison.

Another exemplary aspect of the present disclosure includes a systemincluding an expertise model building module that estimates a targetlevel of knowledge for a topic based on source data and an activity of afirst entity relating to the source data; a user model building modulethat estimates a present level of knowledge of the topic for a secondentity based on the source data and an activity of the second entityrelating to the source data; and a processor that determines whether agap exists between the present level of knowledge of the second entityand the target level of knowledge by executing instructions to comparethe activity of the first entity and the activity of the second entity.

Another exemplary aspect of the present disclosure includes a systemincluding a processor that executes instructions to determine arelationship between a first entity and content relating to a firsttopic and a relationship between a second entity and the contentrelating to the first topic; and an expertise assessment and learningrecommendation module that recommends certain content relating to thefirst topic to the second entity based on the relationship between thefirst entity and the content relating to the first topic and therelationship between the second entity and the content relation to thefirst topic.

Another exemplary aspect of the present disclosure includes a computerprogram product for recommending an activity to a learning user having alevel of expertise of a topic that is less than a target level ofexpertise of the topic, the computer program product comprising acomputer readable storage medium having computer readable program codeembodied therewith. The computer readable program code includes computerreadable program code configured to determine, whether each user of aplurality of users has engaged in an activity relating to any contentassociated with a topic and, if a user has engaged in the activity, todetermine which content the activity is associated with; computerreadable program code configured to determine the target level ofexpertise based on the activity of one or more users having beenidentified as having a level of expertise of the topic that is greaterthan or equal to a certain level of expertise of the topic; and computerreadable program code configured to recommend certain content associatedwith the topic to the learning user based on a comparison of theactivity of the learning user and the activity of the one or more usershaving been identified as having the level of expertise of the topicthat is greater than or equal to the certain level of expertise.

Another exemplary aspect of the present disclosure includes a methodincluding generating a global topic model based on a set of data that isupdated according to an activity of each user of a plurality of users,the global topic model including a topic representation for a topic;generating a plurality of user models, each user model being generatedbased on the activity of a respective user; generating an expertisemodel for the topic based on the activity of at least one user of theplurality of users, the expertise model for the topic setting a targetlevel of knowledge for a first user of the plurality of users; comparinga user model of the first user with the expertise model for the topic,the comparing being performed by a processor of a computer system; andrecommending an activity associated with the set of data to the firstuser based on the comparison.

Another exemplary aspect of the present disclosure includes a methodincluding estimating a target level of knowledge for a topic based onsource data and an activity of a first entity relating to the sourcedata; estimating a present level of knowledge of the topic for a secondentity based on the source data and an activity of the second entityrelating to the source data; comparing the activity of the first entitywith the activity of the second entity; and determining whether a gapexists between the present level of knowledge of the second entity andthe target level of knowledge, based on the comparison of the activityof the first entity and the activity of the second entity, thedetermining being performed by a processor of a computer system.

Another exemplary aspect of the present disclosure includes a methodincluding determining a relationship between a first entity and contentrelating to a first topic and a relationship between a second entity andthe content relating to the first topic; and recommending certaincontent relating to the first topic to the second entity based on therelationship between the first entity and the content relating to thefirst topic and the relationship between the second entity and thecontent relation to the first topic, the recommending being performed bya processor of a computer system.

Another exemplary aspect of the present disclosure includes a computerprogram product for recommending an activity to a learning user having alevel of expertise of a topic that is less than a target level ofexpertise of the topic. The computer program product includes a computerreadable storage medium tangibly embodying first program instructions todetermine, whether each user of a plurality of users has engaged in anactivity relating to any content associated with a topic and, if a userhas engaged in the activity, to determine which content the activity isassociated with; second program instructions to determine the targetlevel of expertise based on the activity of one or more users havingbeen identified as having a level of expertise of the topic that isgreater than or equal to a certain level of expertise of the topic; andthird program instructions to recommend certain content associated withthe topic to the learning user based on a comparison of the activity ofthe learning user and the activity of the one or more users having beenidentified as having the level of expertise of the topic that is greaterthan or equal to the certain level of expertise.

Additional features and benefits will be readily apparent from thefollowing descriptions.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other exemplary purposes, aspects and advantages willbe better understood from the following detailed description of anexemplary embodiment of the invention with reference to the drawings, inwhich:

FIG. 1 depicts an exemplary system block diagram of exemplaryembodiments of the present disclosure;

FIG. 2A depicts an exemplary embodiment of the global topic model shownin FIG. 1;

FIG. 2B depicts exemplary activity-based vectors of a user and exemplaryactivity-based vectors of expertise models;

FIG. 3 depicts a block diagram for recommending an activity to a user;

FIG. 4 depicts an exemplary block diagram for determining if a gapexists in a level of knowledge of an entity;

FIG. 5 depicts an exemplary block diagram for recommending an activityto a user, identifying if a gap exists in a level of knowledge of auser, and identifying an expert to address the gap;

FIG. 6 depicts a typical hardware system, which may be used forimplementing the inventive and novel aspects disclosed herein; and

FIG. 7 depicts typical storage media, which may be used to storeinstructions for implementing the inventive and novel aspects disclosedherein.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS OF THE INVENTION

Referring now to the drawings, and more particularly to FIGS. 1-7, thereare shown exemplary embodiments of the systems, methods, and computerprogram products according to the present disclosure.

The present disclosure includes systems, methods and computer programproducts, which facilitate expediting expertise and provide support fora plurality of learning types including, for example, dynamic,exploratory, and informal learning through the use of topic, user andexpertise modeling.

In some exemplary embodiments, a global topic model including derivedtopics is created based on the aggregation of available content (alsoreferred to herein as “source data”) from various sources. The contentmay include, but is not limited to, metadata and unstructured text. Forexample, if the source data includes a paper, then the content caninclude metadata such as, for example, the date of the paper, the authorof the paper, associated user activities, tags, comments, etc. andunstructured text such as, for example, text from the title and body ofthe paper. If, for example, the source data includes a media file, thenthe unstructured text may include a transcription of the media file. Theunstructured text may be used in order to associate certain content withone or more certain topics.

The content and activities associated with a given user is analyzed inorder to create a topic-based model (also referred to herein as a “usermodel”) of the user. The user model may represent the knowledge of theuser, which relates to each of a plurality of topics.

An expertise model for a given topic of the plurality of topics may becreated by aggregating individual user models of identified experts forthe given topic. Experts may be identified manually or by some existingexpertise identification system.

The difference between a value representing the level of expertise of auser and value representing a target level of expertise of the user fora given topic is determined by comparing the user model of the useragainst the expertise model for the topic. Based on this difference,activities can be recommended to the user in order to reduce thedifference.

By knowing the difference between their level of expertise and theirtarget level of expertise, this feature may help users determine theirprogress in getting to their target level of expertise. Further therecommended activities facilitate the users in getting closer to theirtarget level of expertise, based on dynamic recommendations therebysupporting personalized exploratory learning.

In some exemplary embodiments, the expertise score of a user on a topicis made visible to others. This feature of the present invention servesto motivate users (e.g., enterprise users) to make more of their contentvisible to the system thereby increasing their automatically calculatedexpertise score. For example, as is detailed herein, an expertise scoremay be calculated based on a user's public content and activities. Ifthis score is made visible to others, then a user may be motivated toparticipate in more public activities (e.g., sharing content) toincrease their score. Further, if, for example, a user sees they have arelatively low expertise score for a topic which the user believes theyhave relatively high expertise (e.g., user is shown having an expertisescore that is lower than when the user perceives their expertise to be),then the user may also be motivated to participate in more activities(e.g. sharing content) in order to attempt to increase their score. Insome exemplary embodiments, the user models and the expertise models areupdated regularly, in other exemplary embodiments they are constantlyupdated, in still other exemplary embodiments they are updated based ona user input. This updating can incorporate new content and useractivities, which allows for dynamic evaluation and monitoring of thechange (if any) in the expertise levels of specific individual users, aswell as, the aggregate of all contributors including the experts.

The present inventors have recognized that this approach can also beapplied to aggregated populations, such as, users in a department of anenterprise. For example, management or HR professionals can dynamicallyevaluate and monitor a gap between the current and expected expertiselevels within the department for specific topics.

FIG. 1 depicts an exemplary system block diagram of exemplaryembodiments of the present disclosure. A database 105 stores the sourcedata. This source data may include any available content, such as, forexample, articles, blogs, books, journals, documents, records, audiofiles, video files, websites, presentations, presentation materials,training material, e-mail messages, unstructured text, etc.

The source data may also include metadata. The metadata is associatedwith the respective content. The metadata may include information suchas, how the metadata is related to the content, how often the content isaccessed, who accesses the content, the amount of time that the contentwas accessed, the author of the article, a set of keywords describingthe article or its subject, the number of unique users who have accessedthe content, citations to the content, comments associated with thecontent, etc.

The source data may be from one or more sources. For example, thesources may include employee learning tools, online social businesstools, databases, other information repositories, etc. The sources maybe from within a particular enterprise, from outside a particularenterprise, or a combination thereof. The source data provided by usersmay be made available to others or made public, which is beneficial forat least the aforementioned motivational reasons set forth above.

An extracting module 110 extracts information from the source data andmay be integrated into the system as software, a circuit, or acombination thereof. The information extracted by the extracting module110 may include, for example, text and metadata. The extracting module110 may also extract information relating to a user's activitiesassociated with the content (e.g., whether the user read content, wrotecontent, commented on content, cited to content, etc.) from the varioussources gathered within database 105.

For example, if a first user writes an article and a second user readsand comments on the article, then the extracting module 110 can extractthe text and metadata from the article and determine that the first userwrote the article and that the second user read and commented on thearticle. That is, the extracting module 110, can associate structureddata, as well as unstructured data having been processed in order todetermine certain information that indicates that one or more activities(e.g., that first user wrote article and that second user read andcommented on article) of a user is associated with the content.

A global topic model building module 115 creates a global topic model120. The global topic model building module 115 may be integrated intothe system as software, as a circuit, or as a combination thereof. Theglobal topic model 120 may be created based on the content extracted bythe extracting module 110. In some exemplary embodiments, the globaltopic model 120 may be created based the content and associated useractivities extracted by the extracting module 110.

Since the source data may include unstructured information, the globaltopic model building module 115 may process the unstructured informationaccording to any known method of interpreting unstructured information.For example, the global topic model building module 115 may perform anyknown method for data mining or text analytics in order to categorizethe information.

In some exemplary embodiments, the global topic model building module115 performs either one or both of unsupervised and semi-supervised textanalytics over the source data. In some exemplary embodiments, theglobal topic model module 115 performs these text analyticsautomatically.

In other exemplary embodiments, the global topic model module 115performs these text analytics with no human feedback/annotation or withminimal feedback, such as, for example, an input for parameter valuessuch as the number of clusters to generate. As will be appreciated bythose having skill in the art, there various algorithms that performtext analytics and the present invention is not limited to any specificalgorithm.

However, solely by way of non-limiting example, the global topic modelbuilding module 115, in some exemplary embodiments, uses the LatentDirichlet Allocation algorithm (LDA) to automatically estimate twotopic-related probability distributions: topic-specific worddistribution which describes the probability of a word given a topic,and document-specific topic distribution which describes the probabilityof a topic given a document's content. The input to this algorithm maybe, for example, a set of documents and the number of topic clusters tocreate.

FIG. 2A shows an exemplary embodiment where the global topic model 120includes a plurality of different topics 205A and 205B. These topics mayalso be referred to herein as “topic representations of a topic” or somevariation thereof. Each of the topics may contain a plurality ofsub-topics (e.g., 205A1 and 205A2 of topic 205A). The number of topicsand sub-topics included therein, as shown in FIG. 2A, is arbitrary andmore or less topics and sub-topics included therein may be included.

The global topic model building module 115 determines the topics and thesub-topics through topic modeling. A topic may include words thatfrequently occur together. For example, each topic may be representedusing a set of keywords having certain weights. The sub-topics may eachbe represented using a sub-set of the set of keywords of the respectivetopic.

By way of example, if the source data includes an article, then theglobal topic model building module 115 would associate a bag-of-words ofthe article with the keywords of a topic and determine a topic-specificweight for the article. If the weight is greater than or equal to apredetermined threshold, then the global topic model building module 115would associate the article with the topic. The global topic modelbuilding module 115 performs a similar scheme to associate the articlewith a sub-topic within the associated topic.

For example, a topic 205A of “social software” can be made up ofsub-topics including analytics 205A1, visualization and software 205A2,etc. Therefore, when searching for the difference in a valuerepresenting a level of expertise of a given user and a valuerepresenting a level of expertise of experts of a particular topic, thedifference can be calculated by sub-topic, and even by an item within asub-topic (e.g., the difference between a user level of expertise and abag-of-words in a paper within a sub-topic can be calculated).

Referring back to FIG. 1, a storing module 125 stores informationregarding a user's activities as it relates to content of the sourcedata. The storing module 125 also stores information about the topicsfor which each user has selected to increase expertise. The storingmodule 125 may be integrated into the system as software, as a circuit,or as a combination thereof.

The information stored by the storing module 125 may include metadataassociated with the content extracted by the extracting module. Forexample, if a first user writes an article and a second user reads andcites the article, then the storing module 125 would store informationindicating that the first user wrote the article and that the seconduser read and cited the article.

In some exemplary embodiments, the storing module 125 tracks activity(e.g., the informal learning) that a user undertakes. For example, if auser reads a web page, then a user can mark the web page as somethingthat should be included as an activity of the user when the user model145 of the user is updated.

In some exemplary embodiments, the storing module 125 tracks emerginghot topics (e.g., topics, for which, a predetermined number of usershave selected to increase their expertise) based on an aggregate view ofthe topics for which the users select to increase their expertise.Therefore, emerging hot topics can be ascertained and, as a result, auser or an enterprise can to respond more quickly to changing marketneeds.

A user model building module 130 creates and updates a user model 145for each user based on information from the storing module 125 and theglobal topic model 120. That is, for each user, the user model buildingmodule 130 creates a user model 145 using the content and activitiesassociated with the user. The user building module 125 also uses theglobal topic model 120 to determine the topics of the content andactivities associated with the user. The user model 145 of a user isupdated based on refinements to the global topic model 120 andadditional content as it becomes available and logged activitiesassociated with the user.

A user model 145 estimates one user's knowledge of a topic of the globaltopic model 120 based on the user's activity associated with contentrelated to that topic. For example, if, for a relevant topic, a userreads an article, writes an article, posts a blog, comments on any ofthe aforementioned content, etc., then the user model building module130 will update the user model 145 of the user to be indicative of suchactivity for that topic.

That is, based on the content and activity of a user relevant to aparticular topic, the system assumes that the user has some level ofknowledge of that topic. The user model 145 provides an estimate of whatthe user knows about each topic of the global topic model 120 based ontheir activity.

In some exemplary embodiments, the type of the user's activity/contentof a topic factors into the estimated knowledge of the user of thattopic. For example, the user model 145 of a user that authors a paper ona topic may indicate that the user has greater knowledge of the topicwhen compared with a user model 145 a user that has only read thearticle. Similarly, if the article is cited frequently, then the usermodel 145 of the user that authored the article will indicate that theuser is more knowledgeable about the topic, than if the article is citedinfrequently.

The expert identification module 135 identifies experts associated withvarious topics. The expert identification module 135 may be integratedinto the system as software, as a circuit, or as a combination thereof.For each topic of the global topic model 120, the expert identificationmodule 135 determines whether one or more users are experts of thattopic.

In some exemplary embodiments, users who have engaged in a predeterminednumber of activities with content of a particular topic are dynamicallyidentified as experts on that topic. In other exemplary embodiments,experts on a topic are determined by a weight of each activity (e.g.,authored papers, papers cited, papers read, comments, whether thecomments are positive or negative, peer-reviews, whether thepeer-reviews are positive or negative, etc.). Based on the weight ofeach activity, the users may be ranked relative to one another. Usershaving a rank higher than a predetermined threshold are identified asexperts. Users identified as experts may be ranked relative to oneanother.

There is a possibility that a user having been dynamically identified asan expert may not actually be an expert. Therefore, in some exemplaryembodiments, the expert identification module 135 may receive a manualinput 134 to reclassify a user having been classified as an expert as anon-expert. Similarly, some users who are regarded as experts may not bedynamically identified as such. Accordingly, the expert identificationmodule may receive a manual input 134 to reclassify a user having beenclassified as a non-expert as an expert. These features may bebeneficial because some users may mistrust a dynamic rank without manualfeedback. By allowing users to be manually identified as expert ornon-expert, this issue can be resolved.

For each topic model (e.g., 205A and 205B) of the global topic model120, the user model(s) of the expert(s) for the respective topic areaggregated to create an expertise model 150 for that topic.

Specifically, the expertise model building module 140 aggregates theuser models 145 of the experts identified for the given topic in orderto create the expertise model 150 for that topic.

In some exemplary embodiments, each of the user models 145 include, foreach topic, an activity-based vector that is based on the activities andcontent associated with the user.

The activity-based vector represents an estimate of the level ofexpertise of a user for a respective topic. Each activity is weighted bythe activity type (e.g. whether the user read the content, wrote thecontent, downloaded the content, etc.) and the relevance to the topic,popularity, and importance of the associated content. The dimensions ofthe activity-based vector may be activity types, and the value of eachdimension may be the sum of the weights of all activities of theassociated type.

FIG. 2B shows a user's activity-based vector vs. the aggregateactivity-based vector of the collective experts for exemplary topics205A and 205B, respectively. In this example, the user does not have alot of activities related to topic 205A. For ease of understanding, itis assumed that an arbitrary predetermined threshold is set. It isfurther assumed that the distance between the user's activity-basedvector and the experts' activity-based vector is greater than thepredetermined threshold. Accordingly, the distance between theactivity-based vector of the user and the experts' activity-based vectorindicates that there is a gap in the knowledge of the user for topic205A.

In contrast, it is assumed that the distance between the user'sactivity-based vector and the experts' activity-based vector is lessthan the predetermined threshold for topic 205B. Accordingly, thedistance between the user's activity-based vector and the experts'activity-based vector indicates that there is not a gap in the knowledgeof the user for topic 205B. It should be noted that the predeterminedthreshold can be set for the global topic model 120 as a whole (e.g., auniversal threshold), as well as individually for the topicrepresentation for each topic. It should also be noted that a user'sactivity-based vector can be compared with the aggregate activity-basedvector of the collective experts for each sub-topic of a topic. Such acomparison can be performed in a same (or substantially similar) way tothat discussed above and, therefore, a redundant discussion is notincluded.

The activity-based vectors from the user models 145 of the identifiedexperts are aggregated in order to create an activity-based vector ofthe expertise model for that topic. As would be appreciated by onehaving ordinary skill in the art, different functions such as max, sum,etc., can be used for this aggregation.

An expertise assessment and learning recommendation module 155 isprovided to determine, for an individual user or a population, thedistance to targeted expertise, identify gaps in the knowledge,recommend learning activities, and identify certain users as emergingexperts for a topic. The expertise assessment and learningrecommendation module 155 may be integrated into the system as software,as a circuit, or as a combination thereof

The expertise assessment and learning recommendation module 155 comparesthe user model 145 of the user with the expertise model 150 for thetopic. The expertise assessment and learning recommendation module 155uses a vector distance algorithm to calculate the similarity between theuser's activities and the activities users identified as experts. Thevector distance algorithm can be any of the known vector distancealgorithms. As would be appreciated by one having ordinary skill in theart, different distance metrics can be used (e.g., Cosine similarity,Jensen-Shannon divergence, etc.).

The expertise assessment and learning recommendation module 155calculates the distance 160 between the user's activity-based vector forthe topic and the activity-based vector of the expertise model for thattopic. This distance 160 can be used to determine the difference betweenthe user's estimated knowledge of a topic and the estimated knowledge ofthe experts for that topic.

This distance can also be used to determine if a user has any gaps 165in their knowledge of one or more topics. For example, if the distancebetween the user's activity-based vector and the activity-based vectorof the expertise model is above a predetermined threshold for a topic,then the system will identify a gap in the user's knowledge.

Similarly, this distance can also be used to determine if a user has anygaps 165 in their knowledge of one or more sub-topics. For example, ifthe distance between the user's activity-based vector and theactivity-based vector of the expertise model is greater than apredetermined threshold for a sub-topic, then the system will identify agap in the user's knowledge for that sub-topic. This feature allowsusers to decrease the distance between their activity based vector andthat of the expertise model more easily by allowing the user to focus oncertain sub-topics for which they have an identified gap. Further, thisfeature also allows users to identify whether an expert of a topic has agap in their knowledge of a sub-topic of that topic.

Based on the identified gap 165, the expertise assessment and learningrecommendation module 155 can recommend learning activities 170 to theuser so that the user may reduce or close the gaps in their knowledgefor specific topics. For example, the expertise assessment and learningrecommendation module 155 can recommend that the user read contentrelevant to those topics, identify other users not having the same gapor gaps in their knowledge, suggest other activities based on theactivities of topic experts or users not having the same gap or gaps asthe user. By identifying other users who do not have the same gap orgaps in their knowledge as the user, the expertise assessment andlearning recommendation module 155 is able to recommend a moreknowledgeable user who may be able to answer any questions that the usermay have regarding the topic or sub-topic thereof.

That is, the expertise assessment and learning recommendation module 155recommends activities to help facilitate users in getting closer totheir target expertise level. The candidate activities may include theactivities of the identified experts for the topic, or new activitiesrelated to the content associated with the activities of these experts.The expertise assessment and learning recommendation module 155 rankscandidate activities using a metric that measures the expected return oninvestment, which is calculated based on the expected change in thedifference between the user's current level of expertise and the user'starget level of expertise, the popularity and importance of theassociated content, if any, the similarity or dissimilarity of thecandidate activity with the existing activities of the user, and theestimated cost of performing the candidate activity (e.g. based on itscontent size/length). As a result, the expertise assessment and learningrecommendation module 155 is able to dynamically recommend learningactivities 170 (e.g., materials and tasks) based on a plurality offactors.

Further, by recommending learning activities 170, the exemplaryembodiments of the present disclosure are able to provide a learningpath to raising expertise that is highly customizable and able to suitthe needs and interests of each of the particular users. Moreover, therecommended learning activities direct the user to refined portion ofthe available content thereby helping users more easily reach theirtarget level of expertise.

For example, since the expertise assessment and learning recommendationmodule 155 compares the user's activities with the activities of theexperts, it is able to recommend content and learning activities 170 tothe user, which are the same or similar to those of one or more of theexperts. Therefore, a user is able to follow the path of the experts ofa topic. As a result, when compared to conventional technologies, thisfeature allows for a faster acceleration of expertise of the user.

Since the expertise assessment and learning recommendation module 155compares, for example, any one or more of the following, what the userhas read with what the experts have read, what the user has written withwhat the experts have written, whom the user cites with whom the expertscite, etc., it is able to provide a highly customizable learning paththat is able to suit the needs and interests of each particular user.

The expertise assessment and learning recommendation module 155 is ableto identify certain users as emerging experts 175 for a topic. If, forexample, a user who was not manually or dynamically identified as anexpert 135 (as described above) is determined to have a sufficientlysmall distance (e.g., less than or equal to a predetermined threshold)between their user model 145 and the expertise model 150 for a topic,then the expertise assessment and learning recommendation module 155 maydynamically identify the user as an emerging expert 175 for the topic.If a user is identified as an emerging expert, then in some exemplaryembodiments, the global expertise model 150 for that topic is updated toinclude the user model 145 of the emerging expert.

Further, the expertise assessment and learning recommendation module 155is able to identify certain users as experts of a sub-topic. That is,even if a user is not an expert of a topic, the user may be an expert ofa certain sub-topic within that topic. This feature may be particularlyuseful for identifying experts of esoteric type sub-topics or foridentifying who are knowledgeable about specialized sub-topics within amore general topic. In addition, this feature may allow an enterprise toassemble a team of users that are best able to address a specializedissue or to explain a sub-topic to a user having a particular gap intheir knowledge of a topic.

In some embodiments of the present invention, the topics that aredetected (sometimes automatically) are used as topics for learning,however, if there are one or more key (certain or specific) skills thatare desired to be increased, a mapping can be done between the skill andthe derived topic or topics. This feature may be particularly useful onan enterprise-level for identifying users with knowledge indicating theyhave the key skill, as well as identifying content that is directedtoward the mapped topic (or sub-topic thereof).

In some exemplary embodiments, the comparison between the user model 145and the expertise model 150 can also be applied to aggregatedpopulations. This feature identifies gaps in expertise across thepopulation. It also identifies emerging experts within the population.This feature may be particularly useful in supporting enterprise-levelapplications. For example, if a team of users has a gap in theirexpertise, then the population can be analyzed to determine whether someother user has the requisite expertise. If so, then that user can fillin the gap of knowledge while the team is learning that topic. Thisfeature supports just-in-time expertise sharing.

In some exemplary embodiments, if the number of users having anidentified gap 165 in a particular expertise is greater than (or equalto) a predetermined threshold, then the system will identify a gap inthe expertise level within an enterprise.

Similarly, in some exemplary embodiments, based on the comparison of theuser model 145 of a particular user and the expertise model 150 for aparticular topic, the progress of a particular user can be tracked. Forexample, if the comparison at a first point in time shows that thedistance between the user and the targeted expertise is X and thecomparison at a second point in time shows that the distance between theuser and the targeted expertise is Y, then the relationship between Xand Y can be used to determine the user's progress from the first pointin time to the second point in time.

FIGS. 3-5 show exemplary block diagrams of the exemplary embodimentsdiscussed above. One having ordinary skill in the art would understandthat the block diagrams shown in FIGS. 3-5 can be readily combined whereappropriate. Further the features shown in FIGS. 3-5 can be embodied bysoftware including one or more programs or software modules, one or morecircuits, or any combination thereof.

FIG. 3 shows a block diagram 300 of some of the exemplary embodiments ofthe present disclosure. At 305, a global topic model 120 is generatedbased on the source data 105. At 310, user models 145 are each generatedbased on a respective user's activity associated with the source data105 and the global topic model 120. At 315, the expertise model 150 isgenerated. At 320, one or more user models 145 are compared with theexpertise model 150. At 325, an activity is recommended to a user basedon the comparison at 320 in order to increase the knowledge of the userfor a certain topic. Each activity can include one or more activities.

FIG. 4 shows a block diagram 400 of some of the exemplary embodiments ofthe present disclosure. At 405, an activity of a first entity isdetermined based on the source data 105. As used herein an “entity” mayinclude, for example, one or more of a person, a university, a company,an organization, any combination thereof, etc. An entity may also bereferred to herein as a “user”.

At 410, an activity of a second entity is determined based on the sourcedata 105. Each of the first and second entity can include one or moreentities. The activities are compared at 415 and can each include one ormore activities. Based on the activity of the first entity a targetlevel of knowledge for a topic (and/or sub-topic(s)) is determined at420. Based on the activity of the second entity a present level ofknowledge for a topic (and/or sub-topic(s)) is estimated at 430. At 425,a determination of whether a gap exists between the present level ofknowledge estimated at 430 and the target level of knowledge determinedat 420 can be made according to the comparison of the activitiescalculated at 415.

As explained above, certain activities can be recommended based on thegap in knowledge. These certain activities can be based on theactivities of more knowledgeable entities and can be expected toexpedite the learning of the entity having the gap in their expertise.

Moreover, in a group situation in which one or more of the entities of agroup have a calculated gap in their expertise, one or more entities nothaving the aforementioned gap in the expertise can be added to the groupin order to provide a more well rounded group. Similarly, the otherentities can simply be put in contact with the entities having the gapin order to answer any questions that may arise regarding topics wherethe gap is present.

FIG. 5 shows some exemplary embodiments of the present disclosure. At505, each user's activities are analyzed in order to determine whetherone or more of those activities are associated with the source data 105(or certain content thereof). If a user is determined to have engaged inthe activity, the content of the source data to which that activity isassociated is determined. Based on this determination, a level ofexpertise of each user is determined for each topic and sub-topicthereof of the content of the source data 105. As new content of thesource data is added, these levels of expertise are recalculated.

As a result it is possible to continually ascertain a user's level ofexpertise over a broad range of areas. This allows users (andenterprises) to continually stay abreast of changing market conditions.

At 515, a determination of whether a user's level of expertise isgreater than or equal to a predetermined threshold is made. If the levelof expertise is not at this level, then the user may be classified as alearning user (or a non-expert). If the level of expertise is at orabove this level, then a user may be classified as an expert user.

A learning user for one topic may be considered to be an expert user foranother topic. Similarly, a learning user for one sub-topic of a topiccan be considered to be an expert user for another sub-topic of thatsame topic. Moreover, users may be manually reclassified at thediscretion of an administration of the system.

For each topic (and sub-topic thereof) a user is classified as either alearning user at 520 or an expert user at 525. The activities of therespective users are stored relative to the content of each topic (andsub-topic thereof).

At 530 the activities of one or more learning users for a particulartopic (and/or sub-topic thereof) are compared with the activities of oneor more the expert users for the particular topic (and/or sub-topicthereof). Based on this comparison, at 535, certain learning activitiesexpected to increase the level of expertise of a learning user can berecommended. Also at 535, gaps in the expertise of the learning user(s)can be identified based on the comparison.

At 540, it is shown that if a learning user (or group thereof) has a gapin their expertise, then the system can identify a user not having thatparticular gap in their expertise in order to establish a relationshipthat may close the gap in a group setting or facilitate an expeditedincrease in the level of knowledge for the leaning user(s) having thegap.

As will be appreciated by one skilled in the art, aspects of the presentinvention may be embodied as a system, method or computer programproduct. Accordingly, aspects of the present invention may take the formof an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.) or anembodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module” or “system.”Furthermore, aspects of the present invention may take the form of acomputer program product embodied in one or more computer readablemedium(s) having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for aspects of thepresent invention may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

Aspects of the present invention are described below with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. The flowchart and block diagrams in the Figuresillustrate the architecture, functionality, and operation of possibleimplementations of systems, methods and computer program productsaccording to various embodiments of the present invention. In thisregard, each block in the flowchart or block diagrams may represent amodule, segment, or portion of code, which comprises one or moreexecutable instructions for implementing the specified logicalfunction(s). It should also be noted that, in some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts, or combinations of special purpose hardware andcomputer instructions.

FIG. 6 illustrates a typical hardware configuration 600 which may beused for implementing the inventive cursor control system and method ofcontrolling a cursor. The configuration has preferably at least oneprocessor or central processing unit (CPU) 610. The CPUs 610 areinterconnected via a system bus 612 to a random access memory (RAM) 614,read-only memory (ROM) 616, input/output (I/O) adapter 618 (forconnecting peripheral devices such as disk units 621 and tape drives 640to the bus 612), user interface adapter 622 (for connecting a keyboard624, mouse 626, speaker 628, microphone 632, and/or other user interfacedevice to the bus 612), a communication adapter 634 for connecting aninformation handling system to a data processing network, the Internet,an Intranet, a personal area network (PAN), etc., and a display adapter636 for connecting the bus 612 to a display device 638 and/or printer639. Further, an automated reader/scanner 641 may be included. Suchreaders/scanners are commercially available from many sources.

In addition to the system described above, a different aspect of theinvention includes a computer-implemented method for performing theabove method. As an example, this method may be implemented in theparticular environment discussed above.

Such a method may be implemented, for example, by operating a computer,as embodied by a digital data processing apparatus, to execute asequence of machine-readable instructions. These instructions may residein various types of storage media.

Thus, this aspect of the present invention is directed to a programmedproduct, including storage media tangibly embodying a program ofmachine-readable instructions executable by a digital data processor toperform the above method.

Such a method may be implemented, for example, by operating the CPU 610to execute a sequence of machine-readable instructions. Theseinstructions may reside in various types of storage media.

Thus, this aspect of the present invention is directed to a programmedproduct, including storage media tangibly embodying a program ofmachine-readable instructions executable by a digital data processorincorporating the CPU 610 and hardware above, to perform the method ofthe invention.

This storage media may include, for example, a RAM contained within theCPU 610, as represented by the fast-access storage for example.Alternatively, the instructions may be contained in another storagemedia, such as a magnetic data storage diskette 700 or compact disc 802(FIG. 7), directly or indirectly accessible by the CPU 610.

Whether contained in the computer server/CPU 610, or elsewhere, theinstructions may be stored on a variety of machine-readable data storagemedia, such as DASD storage (e.g., a conventional “hard drive” or a RAIDarray), magnetic tape, electronic read-only memory (e.g., ROM, EPROM, orEEPROM), an optical storage device (e.g., CD-ROM, WORM, DVD, digitaloptical tape, etc.), paper “punch” cards, or other suitable storagemedia. In an illustrative embodiment of the invention, themachine-readable instructions may comprise software object code,compiled from a language such as C, C++, etc.

According to the disclosure herein, one having ordinary skill in the artwill readily appreciate that the present disclosure enables expertiseramp-up by supporting personalized exploratory learning, helping usersunderstand the distance to their target level of expertise, a path totargeted expertise, and their progress in getting to the target. Theseexemplary features alone or in combination facilitate users in gettingcloser to their target with dynamic recommendations for learningmaterials and tasks, support just-in-time expertise sharing in theenterprise by surfacing groups of people for real-timecommunication/questions on a given topic, track the informal learningthat user undertakes, provide a vetting system for evaluating emergingexperts based on automated calculations, identify and rate emergingexperts, identify gaps in expertise on certain topics and certainsub-topics included therein, identify emerging hot topics, and identifythe content that cultivates interaction between experts (via monitoringof content consumed and created by the various users).

While the invention has been described in terms of several exemplaryembodiments, those skilled in the art will recognize that the inventioncan be practiced with modification within the spirit and scope of theappended claims.

Further, it is noted that, Applicant's intent is to encompassequivalents of all claim elements, even if amended later duringprosecution.

What is claimed is:
 1. A method, comprising: generating, as executed bya processor on a computer, a plurality of topic-specific user knowledgemodels for each user of a plurality of users, each topic-specific userknowledge model representing a level of knowledge possessed by arespective user on a single topic from a set of globally defined topicsshared among the plurality of users; generating a plurality oftopic-specific expert knowledge models, each topic-specific expertknowledge model representing an aggregate level of knowledge possessedby a plurality of expert users on a single topic from a set of globallydefined topics shared among the plurality of users; comparing thetopic-specific user knowledge model of the first user with thetopic-specific expert knowledge model for a respective topic todetermine a distance between a user knowledge level and an aggregateexpert knowledge level for the topic, the comparing being performed bythe processor on the computer; and recommending an activity including atleast one of reading, writing, downloading, watching, and commenting ofa content item relevant to the topic to the first user based on saidcomparing.
 2. The method according to claim 1, wherein, in therecommending, the activity includes said reading, said writing, saiddownloading, said commenting, and said watching.
 3. The method accordingto claim 1, wherein said each of the topic-specific user knowledgemodels comprises a topic-specific user activity vector, where eachdimension of the vector represents the user's knowledge level asreflected in a certain type of a user's activities including saidreading, said writing, said downloading, said watching, and saidcommenting of the respective user on the respective topic.
 4. The methodaccording to claim 3, wherein calculation of the topic-specific useractivity vector comprises: combining a topic relevance, a popularity,and an importance of a content item associated with a single useractivity of the respective user to calculate a topic-specific score ofthe activity.
 5. The method according to claim 4, wherein saidcalculation of the topic-specific user activity vector furthercomprises: summing up the topic-specific scores of all user activitiesof the respective user that are of a certain activity type to determinea topic- and type-specific score of a respective activity type for therespective user.
 6. The method according to claim 5, wherein saidcalculation of the topic-specific user activity vector furthercomprises: applying a type-specific weight to the topic- andtype-specific score of the respective activity type to determine a valueof a respective dimension in the user activity vector.
 7. The methodaccording to claim 1, wherein the topic-specific expert knowledge modelcomprises a topic-specific expert activity vector.
 8. The methodaccording to claim 7, wherein calculation of the expert activity vectorcomprises: selecting user activity vectors of the plurality of users whoare determined as experts of the respective topic.
 9. The methodaccording to claim 8, wherein said calculation of the expert activityvector further comprises: aggregating values for a certain dimension ofthe selected user activity vectors to calculate a value of a respectivedimension for an expert activity vector.
 10. The method according toclaim 9, wherein said aggregating comprises calculating summation,maximum, minimum, and mean values for the certain dimension.
 11. Themethod according to claim 1, wherein the distance between the userknowledge level and the aggregate expert knowledge level for the singletopic is calculated based on a vector distance of a topic-specific useractivity vector and a topic-specific expert activity vector.
 12. Themethod according to claim 1, wherein the recommended activity includes acertain type of activity to be performed on a content associated withactivities of the expert users.
 13. The method according to claim 12,wherein a ranking of the recommended activity is determined based on anexpected return-on-investment value whose calculation comprises: anexpected change in a user's current knowledge level if the user engagesin the recommended activity.
 14. The method according to claim 13,wherein the calculation of the expected return-on-investment value isbased on a popularity and an importance of an associated content. 15.The method according to claim 14, wherein the calculation of theexpected return-on-investment value is further based on a similarity ora dissimilarity of the recommended activity with existing activities ofthe user.
 16. The method according to claim 15, wherein the calculationof the expected return-on-investment value is further based on anestimated cost of performing the recommended activity.
 17. The methodaccording to claim 1, further comprising: updating the topic-specificuser knowledge model of the first user after the first user engages in arecommended activity for the topic.
 18. The method according to claim17, wherein the updating comprises: combining a topic relevance, apopularity, and an importance of a content item associated with anaccomplished recommended activity of the respective user to calculate atopic-specific score of the activity.
 19. The method according to claim18, wherein the updating further comprises: applying a type-specificweight and the topic-specific score of an activity to determine a topic-and type-specific score of the activity.
 20. The method according toclaim 19, wherein the updating further comprises: adding the topic- andtype-specific score of the activity to a respective dimension in atopic-specific user activity vector based on a type of the accomplishedrecommended activity.